Skip to content

feat(search): client side fts - #22

Open
larbish wants to merge 29 commits into
mainfrom
feat/client-side-fts-search
Open

feat(search): client side fts#22
larbish wants to merge 29 commits into
mainfrom
feat/client-side-fts-search

Conversation

@larbish

@larbish larbish commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Move search to client-side full-text search: a web worker owns a browser comark-content instance (sqlite-wasm FTS5) hydrated from per-commit snapshot artifacts, replacing the search-sections endpoint that shipped a flat section list on every page hydration.

Built on comarkdown/comark-content#109 (persisted snapshot & manifest artifacts): the webhook parses once per content push and update the cached artifactfs so consumers reuse the cached version.

How it works

Client.

useSearch pins hydration to the production commit via the new /api/content/head endpoint, then a worker fetches /api/content/blob/<sha>/manifest.json and /snapshot/content.json (SHA-pinned, immutable, isr: true cached at the CDN forever) and builds the FTS index through the plugin's cache-first path.

Queries run in the worker: BM25 ranking, heading-weighted scores, snippets.

Server.

The webhook's cache warm becomes warmSnapshot(): one full parse per push, which persists the snapshot artifact into the per-SHA Runtime Cache namespace. The first request on a new SHA is a single cache read, then ISR pins the URL; old SHA URLs simply become unreachable, so search needs no purges. Head-of-branch /blob/<sha> requests reuse the shared prod instance instead of minting a duplicate preview instance. The search-sections plugin, endpoint, route rules, and per-page payload are deleted.

Debug mode

On by default in dev; in production via ?debug=search. Both threads log onto one page-relative timeline — fixed-width lines showing stage durations, decoded/wire sizes, and result counts:

[search]    +412ms  manifest              14.2ms     1.3 KB  1.0 KB on the wire
[search]   +1284ms  index:built          118.4ms             net of the snapshot fetch above
[search]   +1290ms  index:rows                                87 results · sections in the FTS table

Next steps to discuss

  • Extract the search stack into a plugin — useful for nuxt.com (migration on comark-content in progress).

  • Version-aware search — the hydration URLs are already per-SHA (/api/content/blob/<sha>/…), so searching any historical version is just pointing the worker at that commit's artifacts, which the /blob/<sha> routes already serve and self-heal on first request. Would pair with the existing version-history UI; main open question is worker lifecycle when switching versions (one instance per SHA vs. reset + rehydrate).

@vercel

vercel Bot commented Aug 13, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
comark-docs-layer Ready Ready Preview Sep 7, 2026 4:27pm UTC
comark-docs-playground Ready Ready Preview Sep 7, 2026 4:27pm UTC

@pkg-pr-new

pkg-pr-new Bot commented Aug 19, 2026

Copy link
Copy Markdown
npm i https://pkg.pr.new/comarkdown/comark-docs@22

commit: 83fb86f

@larbish
larbish requested review from atinux and farnabaz August 19, 2026 16:58
@larbish
larbish marked this pull request as ready for review August 19, 2026 16:59
Comment thread app/error.vue
@coldtea-pr-lens

coldtea-pr-lens Bot commented Sep 3, 2026

Copy link
Copy Markdown

◈ PR Lens

🟢 +3 new · 🟠 ~12 changed · 🔴 -0 removed · 3 flows · 30 files · commit 83fb86f


Architecture

Architecture diagram for comarkdown/comark-docs at 83fb86f

15 components touched across 6 lanes.

Open the interactive canvas


Inside the changed components — 3 views

Component view — Client search

The client-side search worker running SQLite WASM FTS5 hydrated by CDN snapshot artifacts

Architecture view of Component view — Client search in comarkdown/comark-docs

Component view — Build snapshots & cold starts

Build-time snapshot generator and runtime hydration from bundled Nitro server assets

Architecture view of Component view — Build snapshots & cold starts in comarkdown/comark-docs

Component view — Webhook revalidation

Manifest-based change detection, search artifact warming, and targeted ISR cache purges

Architecture view of Component view — Webhook revalidation in comarkdown/comark-docs

Data flow

Data flow diagram for comarkdown/comark-docs at 83fb86f

Client-side search hydration and query · Push webhook revalidation and warming · Cold start hydration from snapshot

Open the interactive canvas


The other flows — 2 sequences

Push webhook revalidation and warming

Sequence diagram of Push webhook revalidation and warming in comarkdown/comark-docs

Cold start hydration from snapshot

Sequence diagram of Cold start hydration from snapshot in comarkdown/comark-docs

Drill down
Client & Browser — 4 components
🟡 CHANGED App Shell & Search UI

Renders application layout and mounts the new AppSearch palette connecting to the search worker.

🟢 NEW SQLite Search Worker

Runs an in-browser SQLite WASM FTS5 search index hydrated from immutable CDN snapshot artifacts.

🟡 CHANGED useDocsContent

Client content client wrapper with obsolete search-sections plugin removed.

🟡 CHANGED Assistant Chat

AI chat drawer rendered in the documentation layout.

Edge & Routing — 2 components
🟡 CHANGED Edge ISR Cache

Vercel edge route cache configured with immutable rules for per-commit search artifacts.

🟡 CHANGED Markdown Content Negotiator

Rewrites raw markdown requests for agents and automated clients.

Server Routes & APIs — 4 components
🟡 CHANGED Content API

Dispatches content requests, exposes head commit SHA, and serves search snapshot artifacts.

🟡 CHANGED Revalidate Webhook

Receives GitHub push webhooks, diffs manifest keys, warms search artifacts, and purges ISR cache.

🟡 CHANGED Raw Markdown Route

Serves raw markdown page mirrors using updated navigation utilities.

🟡 CHANGED Code Explorer API

Serves example source code with syntax highlighting using updated theme helpers.

Content Engine — 2 components
🟢 NEW Snapshot Generator

Resolves git commit SHA at build time and bundles parsed markdown JSON into Nitro server assets.

🟡 CHANGED comark-content Engine

Loads pre-parsed snapshot from Nitro server assets on cold start, falling back to GitHub API.

Cache & Datastores — 2 components
🟢 NEW Nitro Server Assets

Stores bundled JSON content snapshots inside the serverless deployment package.

🟡 CHANGED Ref Pointer Cache

Caches resolved git commit SHAs for branch heads and content directories.

External Services — 1 component
🟡 CHANGED GitHub API

Provides content trees, commit history, and push webhook events.


View

  • Architecture lens
  • Data flow lens
  • Expand every detail
  • Show unchanged neighbours

Tip

Would you rather run it from CI on a key of your own? Add .github/workflows/pr-lens.yml with coldteadotai/pr-lens/packages/action@v0 and a model key in your repository secrets, say GEMINI_API_KEY. The Action asks Gemini by default, or OpenAI and any endpoint speaking /chat/completions through its provider input.

🪧 More tips
  • Run PR Lens on your own machine: npx skills add coldteadotai/pr-lens installs the agent skill. Then tell your coding agent: "Diagram the change you just made with PR Lens and attach it to the pull request."
  • Draw a diff before it is even a pull request: npx @coldtea/pr-lens-cli analyze --base origin/main reads the diff with your own model key, and npx @coldtea/pr-lens-cli render .pr-lens/graph.json draws the same lenses on your machine.
  • The boxes under View are live. Tick Architecture lens or Data flow lens to choose which diagrams appear, or Expand every detail to open every drill-down at once. The comment redraws in place a few seconds later.
  • Show unchanged neighbours lists the components this change did not touch alongside the ones it did, so the drill-down shows what the changed code sits next to.
  • GitHub will not let you zoom an image in a comment. The link under each diagram opens it on an interactive canvas, where you can zoom, pan and step through the flow.
  • The CLI's render picks up .github/pr-lens.yml automatically and applies your corrections (renames, exclusions, lane pins) at draw time.
  • PR Lens is free for open source. A star on the repository is what keeps it going.
  • Push a new commit and the whole comment re-renders for the new head. An older run never overwrites a newer one, so a slow render cannot put a stale diagram back.
  • The diagrams follow your GitHub theme, so dark mode gets the dark render and light mode the light one, and the moving dots show this pull request's data in motion.

◈ Rendered by PR Lens · crafted with ❤️ by the Coldtea team · Come say hi on Discord

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants